Comparing different machine learning techniques for predicting COVID-19 severity

Background Coronavirus disease 2019 (COVID-19) is still ongoing spreading globally, machine learning techniques were used in disease diagnosis and to predict treatment outcomes, which showed favorable performance. The present study aims to predict COVID-19 severity at admission by different machine learning techniques including random forest (RF), support vector machine (SVM), and logistic regression (LR). Feature importance to COVID-19 severity were further identified. Methods A retrospective design was adopted in the JinYinTan Hospital from January 26 to March 28, 2020, eighty-six demographic, clinical, and laboratory features were selected with LassoCV method, Spearman’s rank correlation, experts’ opinions, and literature evaluation. RF, SVM, and LR were performed to predict severe COVID-19, the performance of the models was compared by the area under curve (AUC). Additionally, feature importance to COVID-19 severity were analyzed by the best performance model. Results A total of 287 patients were enrolled with 36.6% severe cases and 63.4% non-severe cases. The median age was 60.0 years (interquartile range: 49.0–68.0 years). Three models were established using 23 features including 1 clinical, 1 chest computed tomography (CT) and 21 laboratory features. Among three models, RF yielded better overall performance with the highest AUC of 0.970 than SVM of 0.948 and LR of 0.928, RF also achieved a favorable sensitivity of 96.7%, specificity of 69.5%, and accuracy of 84.5%. SVM had sensitivity of 93.9%, specificity of 79.0%, and accuracy of 88.5%. LR also achieved a favorable sensitivity of 92.3%, specificity of 72.3%, and accuracy of 85.2%. Additionally, chest-CT had highest importance to illness severity, and the following features were neutrophil to lymphocyte ratio, lactate dehydrogenase, and D-dimer, respectively. Conclusions Our results indicated that RF could be a useful predictive tool to identify patients with severe COVID-19, which may facilitate effective care and further optimize resources. Graphical Abstract Supplementary Information The online version contains supplementary material available at 10.1186/s40249-022-00946-4.


Background
Coronavirus disease 2019 (COVID-19) has been a severe public health event and is still ongoing spreading globally [1,2]. The main manifestations of COVID-19 include fever, fatigue, and dry cough, most COVID-19 cases are non-severe with favorable outcome [3,4]. However, patients with severe COVID-19 tend to progress rapidly and experience respiratory failure, respiratory distress syndrome, and septic shock or even death within a short period of time [5], with a high mortality of 53% [6]. Previous studies observed that male, aged over 65, smoking, elevated lactate dehydrogenase, elevated D-dimer, and chest imaging findings associated with COVID-19 severity [7][8][9][10]. However, the studies didn't demonstrate indicators' contribution to COVID-19 severity on admission. Hence, it is essential to assess importance of features to disease severity on admission for earlier and more targeted care, further reduce severe disease and prioritize medical resource.
Machine learning as an effective and innovative tool has been reported applications in diabetes [11], cardiovascular diseases [12], cancer [13,14], sepsis [15], and depression [16], etc., and showed favorable performance. Moreover, machine learning was used to predict COVID-19 diagnosis and treatment outcome, etc. [17][18][19][20][21]. However, different machine learning techniques were used to predict COVID-19 severity have not well been reported. Our study aimed to compare different machine learning techniques including random forest (RF), support vector machine (SVM), and logistic regression (LR) to predict COVID-19 severity based on clinical, laboratory, and chest computed tomography (C-CT) features, further identify feature importance to COVID-19 severity. The results can provide reference for clinicians to diagnose and treat patients timely and effectively, and further optimize medical resources, especially in resource-limited areas.

Study design and participants
A retrospective study was conducted in the JinYinTan Hospital from January 26 to March 28, 2020. Data of demographic, clinical, laboratory, C-CT features, and treatment outcome were reviewed from 287 patients of hospital information system, all cases were laboratoryconfirmed by real-time reverse transcriptase polymerase chain reaction assay.

Feature selection
In the present study, 30 features were remained after the initial 86 features were screened by LassoCV in the study. Subsequently, we carried out spearman's rank correlation

Graphical Abstract
Comparing different machine learning techniques for predicƟng COVID-19 severity Feature importance ROC curves of RF, SVM, and LR The heat map of correlaƟon between features to further reduce confounding, features with correlation coefficients greater than 0.4 (r > 0.4)were excluded. For example, white blood cell (WBC) was correlated with neutrophil to lymphocyte ratio (NLR); red blood cell (RBC) and hematocrit (HCT) were both correlated with hemoglobin concentration (HB); albumin (ALB) is associated with cholinesterase (CHE), etc. Additionally, combined with expert opinions, and literature evaluation 7 features were excluded from 30 features, namely WBC, RBC, HCT, ALB, r-glutamyl transpeptidase(r-GT), CHE, creatine kinase (CK), then 23 features were remained ( Fig. 1), correlation coefficient between features were less than 0.4 (Fig.2), which were entered into models for further analysis.

Features and models
Both dichotomous features are containing precisely two distinct values in the form of 0 and 1, for the three classification features such as C-CT, 0, 1 and 2 as three categorical variables were used to represent no lesions, unilateral lesions, and bilateral lesions, respectively; continuous variables were expressed with unit standardization. The 23 features were processed by RF, SVM, and  The heat map of correlation between features. Color indicates the value of the correlation coefficient (r). The color intensity is proportional to the correlation coefficient (r), with positive correlations (r > 0) shown and negative correlations (r < 0), lower intensive color indicates lower correlations, in the study, 23 features selected by machine learning techniques, each feature is weakly correlated with each other (r < 0.4). Twenty-three features including C-CT, fever, MT, HR, SBP, HB, NLR, RDW-CV, IMG, IL-6, ESR, IBIL, ALP CysC, LDH, Ca 2+ , AFU, AMS, RBP, Hs-cTn, MB, PT(s), D-dimer. C-CT chest computed tomography, MT malignant tumor, HR heart rate, SBP systolic blood pressure, NLR neutrophil-to-lymphocyte ratio, HB hemoglobin concentration, RDW-CV red cell volume distribution width, LDH lactate dehydrogenase, IBIL indirect bilirubin, PT prothrombin time, ESR erythrocyte sedimentation rate, AFU α-fucosidase, RBP retinol-Binding protein, IL-6 interleukin-6, Hs-cTn hypersensitive troponin, AMS amylase, CysC cystatin C, IMG immature granulocyte, ALP alkaline phosphatas, MB myoglobin LR modeling to compare the three models in predicting COVID-19 severity on admission.
RF is a classifier that contains multiple decision trees, and the output is determined by the voting results of multiple trees. We used a grid search method to determine the number of classifiers, maximum depth, and minimum number of branch samples in random forest modeling. SVM based on statistical learning theory, is a class of generalized linear classification technique that classifies data binary in a supervised learning manner, and correctly classifies samples by looking for decision boundaries. LR is a generalized linear regression analysis model that uses different kernel functions to estimate data conditions. Similar to random forest, we use grid search method to determine the selection of key parameters of SVM and LR, thereby reducing the training error in modeling.

COVID-19 severity for prediction
COVID-19 severity was determined based on the Diagnosis and Treatment Protocol for COVID-19 (seventh edition) [22]. In our study, given the small sample of 287 patients analyzed from single center, therefore, COVID-19 cases were categorized as non-severe and severe cases. Mild and moderate cases were classified as non-severe cases, severe and critical were classified as severe cases. The outcome variable was coded as binary 0 or 1, and the predictive output variable was defined as severe type (coded as 1) and non-severe type (coded as 0).

Model training and generalization
Of three models, 10-fold cross-validation was used to determine best appropriate data set to further better accuracy on predictions. Firstly, we split the all data into 10 number of folds. Each model has been trained and tested 10 times. For every iteration we selected 1-fold as test data and rest all as training data and ensured that different sets of data were used for model training and testing each time. Finally, we summed up the score for every fold and took the average to get the overall score.

Evaluation of model performance
The model's performance was evaluated respectively to compare the sensitivity, specificity, accuracy and area under curve (AUC) of receiver operating characteristic (ROC) of different models for severe COVID-19. The cut-off value was determined by the point on ROC curve where the Youden index was highest.

Importance of test types
In order to quantify the importance of clinical, laboratory, and C-CT features on the predicted outcomes, the best performance model was used to provide relative feature importance by computing the contribution of each input feature toward the predictive outcome.

Statistical analysis
Categorical and continuous variables were expressed as n (%) and median (interquartile range, IQR), respectively. The Mann Whitney U test, χ 2 test, or Fisher's exact test were used to compare differences between severe and non-severe cases, as appropriately. The LR, SVM, and RF models were all performed, and AUC were calculated by using python 3.8.3 (Python Software Foundation, Delaware, USA) and anaconda 4.10.1 (Red Hat, Raleigh, North Carolina, USA). Main python module was shown in Additional file 1.

Cohort characteristics
A total of 287 patients enrolled in our study. The median age was 60.0 years (IQR: 49.0-68.0 years) with 63.0 years (IQR: 55.5-70.0 years) for severe cases and 57.0 years (IQR: 46.7-67.0 years) for non-severe cases. A higher proportion (56.4%) was male, and the proportion of severe and non-severe cases were 36.6% and 63.4%, respectively. The most common symptoms reported were cough (61.3%) and fever (30.7%), 23 features of COVID-19 patients in the cohort are demonstrated in Table 1.

Predictive performance
We further analyzed the predictive performance of RF, SVM and LR with AUC, sensitivity, specificity, and accuracy ( Table 2, Fig. 3), in terms of predictive performance among the three models, we observed that the overall better performance by AUC of 0.970 were RF for predicting COVID-19 severity at admission compared to SVM with AUC of 0.948 and LR with AUC of 0.928. Moreover, RF also achieved a favorable sensitivity of 96.7%, specificity of 69.5%, and accuracy of 84.5%. SVM had sensitivity of 93.9%, specificity of 79.0%, and Table 1 Twenty-three features of COVID-19 patients in the cohort Data were presented median (IQR) or n (%). P values were estimated by comparing variables between severe and non-severe cases a "0" denotes" no lung lesions", "1" denotes "unilateral lung lesions", and "2" denotes "bilateral lung lesions" MT Malignant tumor, HR Heart rate, SBP Systolic blood pressure, NLR Neutrophil-to-lymphocyte ratio, HB Hemoglobin concentration, RDW-CV Red cell volume distribution width, LDH Lactate dehydrogenase, IBIL Indirect bilirubin, PT Prothrombin time, ESR Erythrocyte sedimentation rate, AFU α-fucosidase, RBP Retinol-Binding protein, IL-6 Interleukin-6, Hs-cTn Hypersensitive troponin, AMS Amylase, CysC cystatin C, IMG Immature granulocyte, ALP Alkaline phosphatas, MB Myoglobin, C-CT Chest computed tomography

Feature importance
We further analyzed the top 23 features ranked by relative importance to COVID-19 severity by using RF with best performance. C-CT had highest importance to COVID-19 severity, and the following were NLR, LDH count, and D-Dimer, respectively (Fig. 4).

Discussion
Previous studies observed that RF, LR, SVM, gradient boosted decision tree, and neural network to predict COVID-19 [23][24][25]. Schwab et al. demonstrated that RF had highest predictive performance for predicting hospital admissions for COVID-19 patients, and critical care admissions for COVID-19 cases in terms of AUC compared to LR, SVM, neural network, and gradient boosting [25]. In the present study, we found that RF could predict COVID-19 severity with best performance compared to SVM and LR. The results highlight the possibility that high-performance of RF are able to predict disease severity, the findings can provide reference to monitor parameters effectively and further optimize medical resource.
Chest CT is a widely used imaging test, which can be useful for initial triage of suspected COVID-19 patients. Previous studies demonstrated that chest CT or X-rays findings were associated with COVID-19 severity based on machine learning [26][27][28][29][30], our data observed that chest CT findings with highest importance scores was for diagnosis of severe COVID-19 on admission by RF model, which can help clinicians a fast diagnosis and severity assessment of COVID-19 even before molecular test results are available or laboratory testing is limited. Wang et al. reported that COVID-19 clinical severity could be assessed based on the proposed chest lung-lesion automated deep-learning model [26], given deep learning was likely to be over-fitting for small sample, therefore, which was not used in the study. Following the C-CT ranked feature importance scores were NLR, LDH, D-Dimer, Ca 2+ , ESR, and AFU, respectively, were confirmed in previous studies [30][31][32]. Ponti et al. reported that NLR was closely in relationship with coagulation cascades in disseminated intravascular coagulation and acute respiratory distress syndrome in severe COVID-19 patients [33]. LDH and D-dimer are consistently reported among hospitalized severe COVID-19 patients [7], while gradual development during the disease course is particularly related to disease worsening in adults and children [34,35]. However, compared with previous researches, AFU and HB are rarely mentioned in COVID-19 severity. AFU is essential in viral and bacterial infections including the cell-cell adhesion processes that mediate inflammation [36,37]. Studies illustrated that AFU may be potentially upregulated by chemokines in the later stages of inflammation, such as COVID-19 [37,38]. We also found that AFU is a significant lab indicator to distinguish severe illness, which may be a possible therapeutic target in the future. In other studies, hemoglobin and red blood cell count were found lower in COVID-19 patients, however, with no significant difference between survivors and non-survivors [39]. COVID-19 patients might suffer from hyper-metabolic states, such as septic shock and multiple organ dysfunction syndromes, which needs an increasing capability of hemoglobin to support the peripheral tissue demands for oxygen [40]. The ability to meet the oxygen demands of the peripheral tissues caused by the COVID-19 infection becomes the leading role to decide the prognosis, according to recent evidence [41].
Our study has several limitations. Firstly, the small sample size from a single study site is a major limitation of the present study, which may undermine reliability of our analysis. Secondly, the data was retrospective collected, some data were incomplete or missing (such as smoking and drinking, and educational level), which were not further analyzed in the study. Further large sample, multi-center studies should be conducted to confirm our results.

Conclusions
Our study indicates that RF may be a prioritization tool in predicting COVID-19 severity with favorable overall performance at admission. The prediction model can potentially assist clinicians to facilitate severe COVID-19 patients' identification promptly and apply treatment timely, further optimize potentially healthcare resources.